Deformable registration of computer-generated airway models to airway trees

ABSTRACT

A system for registering a luminal network to a 3D model of the luminal network includes a computing device configured to identify potential matches in the 3D model with location data of a location sensor, assigning one of the potential matches a registration score based on a deformation model applied to the 3D model, and displaying the potential match having the highest registration score.

INTRODUCTION

Minimally-invasive surgical (MIS) procedures are a common method ofperforming various diagnostic and/or treatment procedures on a patient.Among other benefits, MIS procedures pose lower risks and shorterrecovery times to the patient relative to other surgical procedures. OneMIS procedure, Endobronchial Navigation Bronchoscopy (ENB), involvesinserting a bronchoscope and various catheters into a patient's airwaysto navigate one or more tools to a treatment site to perform adiagnostic and/or treatment procedure. ENB is an effective procedure foraccessing areas of the patient's lungs and surrounding parenchyma whileminimizing injury to the patient.

Various systems, devices, and computer-implemented methods have beendeveloped to provide image guidance to assist the clinician inidentifying the location of the tools in the patient's airways. One suchsystem includes generating one or more three-dimensional (3D) models ofthe patient's airways based on image data of the patient's chest anddisplaying a determined location of the tools on the 3D models.

SUMMARY

Provided in accordance with embodiments of the disclosure are methodsfor registering a luminal network to a 3D model of the luminal network.The method includes receiving location data associated with a locationsensor on a tool. The tool navigates the luminal network. The methodalso includes identifying potential matches in the 3D model with thelocation data and assigning one of the potential matches a registrationscore based on a deformation model applied to the 3D model. The methodfurther includes displaying the potential match having a highestregistration score.

In another aspect of the disclosure, the assigning of the registrationscore is further based on a distance between the potential match and thelocation data, and/or a difference in direction between location sensordirection data and a direction of the potential match.

In a further aspect of the disclosure, the method also includes, afterreceiving further location data, reassigning the potential match anupdated registration score based on the further location data.

In yet another aspect of the disclosure, anchor points from the initiallumen registration are used to constrain the registration.

In another aspect of the disclosure, potential matches assigned a lowregistration score are discarded.

In a further aspect of the disclosure, the method also includes delayingdiscarding potential matches with low registration scores when thepotential matches are in a bifurcation area of the 3D model.

In yet another aspect of the disclosure, the method further includesupdating the 3D model with a plurality of potential matches having thehighest registration score.

In another aspect of the disclosure, the deformation model includesidentifying a first region of the luminal network and a second region ofthe luminal network. The deformation model may further include modelingat least one of rotation, compression, extension, and bending for thefirst region and the second region independently. The deformation modelmay also include modeling at least one of rotation, compression,extension, and bending for the first region and the second region withadjacent regions having interdependence. The deformation model mayinclude performing rigid registration of the first and second regions toform first and second rigid registrations and stitching together thefirst and second rigid registrations.

In a further aspect of the disclosure the luminal network is a bronchialairway, and the deformation model includes applying a moving window froma trachea to a periphery of the bronchial airway. The method may includeperforming a rigid registration in the moving window.

In another aspect of the disclosure, the deformation model may includeweighing registration for central areas of the lungs greater thanperipheral areas of the lungs or using deformation models havingdifferent deformation characteristics for at least one of a right lung,a left lung, and each lung lobe. The deformation model may include usinga biomechanical model of the lungs based on finite image analysis, usinga deformation model based on a specific disease, and/or using at leastone of a bronchoscope model and a catheter model to produce a specificdeformation model.

Embodiments of the disclosure also provide a system for updating a 3Dmodel of a luminal network. The system includes a location sensorcapable of being navigated within a luminal network inside a patient'sbody. The system further includes an EM navigation system including anEM field generator configured to generate an EM field, and an EMtracking system configured to detect a location of the EM sensor withinthe EM field. The system also includes a computing device including aprocessor and a memory storing instructions. The instructions, whenexecuted by the processor, cause the computing device to receivelocation data associated with a location sensor on a tool. The toolnavigates the luminal network. The instructions further cause thecomputing device to identify a plurality of potential matches in the 3Dmodel with the location data and assign at least one of the potentialmatches a registration score based on a deformation model applied to the3D model. The instructions also cause the computing device to displaythe potential match having a highest registration score.

A further aspect of the disclosure is directed to a method ofregistering a 3D model to a luminal network. The method includesgenerating a 3D model of a luminal network from a pre-procedure imagedata set, conducting a survey of the luminal network, generating a 3Dmodel of the luminal network from data captured during the survey, andapplying a transform to the 3D model of the luminal network generatedfrom the pre-procedure image data set to approximate the 3D model of theluminal network generated from the data captured during the survey.

In accordance with further aspects of the disclosure, the survey data iscollected with a sensor inserted into each lobe of a lung of a patient.Further, the sensor may be an electromagnetic sensor collectingelectromagnetic position data. Still further aspects of the disclosureinclude applying a plurality of transforms to the 3D model of theluminal network generated from the pre-procedure image data set. Thetransform may be selected by determining which of a plurality oftransforms result in a best fit of the 3D model generated from thepre-procedure image data to the 3D model from the data captured duringthe survey.

Any of the above aspects and embodiments of the disclosure may becombined without departing from the scope of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects and features of the disclosure are described hereinbelowwith references to the drawings, wherein:

FIG. 1 is a schematic diagram of an endobronchial system for planningand performing treatment of an area of a patient's lungs;

FIG. 2 is a block diagram of an example computing device forming part ofthe system of FIG. 1 ;

FIG. 3 shows a flowchart of an example method for registering a luminalnetwork to a 3D model of the luminal network;

FIG. 4 shows an exemplary 3D model of a bronchial airway includingsensor data;

FIG. 5 shows a lumen registration in which the system matches newsamples to an airway of the 3D model;

FIGS. 6A-6D illustrate registration with a 3D model before and afterbranching of the bronchial airway; and

FIG. 7 shows a lumen registration performed using a moving boundary box.

FIG. 8 is a flowchart showing an exemplary method of transforming apre-procedural CT image data set to correspond to survey data.

DETAILED DESCRIPTION

The disclosure is directed to devices, systems, methods, andcomputer-readable media for registering a 3D bronchial tree model(hereinafter referred to as a “3D model”) with a patient's airways basedon electromagnetic navigation.

There is a continuing need for systems and methods that accuratelyregister 3D models to the patient's actual airways. Since lungs areflexible, dynamic, and deformable organs, there is usually asignificant, dynamic difference between the bronchial tree of thepatient during a procedure and the bronchial tree in the CT/MRI image ormodel.

As such, the systems and methods discussed herein accommodate theflexibility and elasticity of the lungs using a set of registrationmethods.

During ENB procedures, it is important to register the magneticnavigation space to the patient's CT scan. As understood in the art,“registration” refers to a translation, mapping, transformation, or thelike, which converts locations or coordinates in one space to locationsor coordinates in another space. In order to perform such registration,several methods have been developed. One method, known as manualregistration, is based on the visual identification of the bronchoscopewith the main airway's branching points and associating them with thepoints visually identified on the CT scans. During the ENB procedure,the user navigates the bronchoscope with a catheter fitted with themagnetic sensing coil to the branching points in the airways, andmanually generates the associations between the branching points and thelocations in the magnetic navigation space. Another method, known asautomatic registration, is based on collection of the point cloud fromthe airways (called survey) using a catheter with the magnetic sensor atits tip, and then fitting the survey to the image of the airway treederived from the CT scan using a suitable image segmentation algorithm.

Due to the flexibility of the lungs, the actual shape of the lungsduring the time of a surgical procedure can be deformed or differentfrom the shape at the time of the CT scan and/or initial registration,resulting in the reduction of navigation accuracy. This deformationcaused by the flexibility of the lungs may be due to severaldifferences, such as: (1) the CT scan being performed while the patientis holding their breath after a full inhale, whereas during the surgicalprocedure, both registration and navigation, the patient is sedated andbreathing normally; (2) the patient may be horizontal for a much longerperiod during the surgical procedure thereby causing increaseddeformation; (3) during the surgical procedure, while the bronchoscopeis inside of the patient, the head, neck, and superior part of the chestmay also be deformed; and/or (4) the CT scan being performed on aconcave bed, as is typical, while the surgical procedure is generallyperformed while the patient is lying flat.

In navigational bronchoscopy (for example electromagnetic navigationbronchoscopy, also referred to as ENB) systems, rigid or semi-rigidalgorithms are often used to map the intra-procedural bronchial tree orairway carinas to the bronchial tree or airway carinas in a CT scan.However, because the lungs are flexible, dynamic, and deformable organs,there is usually a significant, dynamic difference between the bronchialtree of the patient during a procedure and the bronchial tree in the CTscan. While described in the disclosure as a CT scan, other suitableimaging modalities such as, for example, MM, may be used to generateimage data (e.g., MM scan) of the bronchial tree or airway carinas.

The bronchial tree deforms due to respiration, changes in patientposition, the force applied by the bronchoscope and other endoscopictools, and other reasons. The deformation is different for each patientand can be significantly different for the same patient on differentoccasions. Therefore, it is difficult to estimate with sufficientaccuracy how the lungs will deform from the time of the CT scan to thetime of the procedure. However, it has been empirically shown thatcertain deformations are more probable than others. In order to utilizethis information to provide more accurate registration, thesedeformations are modeled. Deformable registration includes aligning thevirtual bronchial tree to a 3D model of the bronchial tree obtainedthrough a CT scan. Each branch is aligned independently, and/or certaindeformations are deemed more reasonable and others less reasonable.

Any such model has a “registration score” or “likelihood score” which ishigher for more likely deformations and lower for less likelydeformations. Regional deformation models can have this score, orconfidence interval, based on what happens in other areas, for instanceadjacent areas, and the deformation model may be selected based on thebest score. Additionally, the model score could be affected by typeand/or characteristic of the tool used.

Lungs, lobes, segments, airways or other regions in the lung can bemodeled to rotate independently of each other, extend, compress, orbend. The likelihood of a particular deformation may be consideredidentical for the entire lungs or may change depending on bifurcationgeneration, inserted catheter, and other parameters. Deformation in aspecific lung, lobe, segment, or airway can be modeled to affectpotential deformations in sub-regions or neighboring lung regions. Forexample, if the right main bronchus has extended 20%, it may be morelikely that the intermediate bronchus has extended as well.

The lungs may be divided into multiple arbitrary or anatomical regions.In each region, a rigid registration is performed, and then allregistrations may be smoothly “stitched” into a single model or image.This process includes separately registering regions based on regionalmodels, which may be different, and then smoothly aligning the multipleregistrations. Smoothing may be accomplished using a suitable imagestitching algorithm.

A moving window may be applied from the trachea to the periphery of thelung.

Within each window, a rigid registration may be performed. Thus,deformation may progressively increase as the image approaches theperiphery of the lung.

In the periphery of the lungs, confidence in registration is reduced byfactors such as, for example, small and close airways, respiration,missing branches in the segmented bronchial tree, and limited surveys.Therefore, regardless of the deformation model that is used,deformations in the central area of the lungs generally may be trustedmore. In addition, if a certain local deformation is not very likely,but there are no other more likely candidates, then this deformation canalso be trusted more than if alternative deformations were apossibility.

There could be different deformation characteristics for the right andleft lung and/or for each of the lung lobes or segments. For example, itmay be empirically shown that the lateral basal segments are oftenrotated to the right, while apical segments compress.

Various biomechanical models of the lungs may be used, including, forexample, models based on finite image analysis. These models may beutilized for predicting respiratory movement during radiotherapy. Theuse of these models could be expanded to provide deformable registrationfor ENB. Certain diseases such as emphysema or various lung lesions andtumors can be modeled to produce specific local or global deformations.Different bronchoscope and catheters models can be modeled to producespecific deformations. Deformation can depend on the mechanicalproperties of the endoscopic tool, its flexion or orientation relativeto the airway wall, the diameter and stiffness of the airway, thedirection of motion (forward or backward), the position of the toolrelative to a bifurcation, and other variables.

Causes for divergence from the CT scan to the patient's body can bemodeled separately and then combined. For example, a certain patientposition during CT scan and a different position during the ENBprocedure (hands up/down, etc.) may be modeled. A different bedcurvature in the CT scanner than in the ENB procedure bed may also bemodeled. The application of force by the bronchoscope and catheter onthe airway walls may be modeled. Sedation, anesthesia, and/or muscleblockers have an effect on lung anatomy, which may be modeled. Likewise,a change in patient anatomy from the time of the CT scan may impactregistration, and may also be modeled (e.g., changes caused by thedevelopment of pneumonia, a change in weight, etc.). In computing thedivergence, the minimum distance from each survey point to a prospectivetransformation of the reference airway tree may be multiplied by theweight assigned to the corresponding survey point, and the weightedminimum distances are averaged. This process for a particular region isreferred to as optimized region registration.

Still other causes for changes in airway pathways include fullinhalation during CT scan and tidal respiration cycle during ENBprocedure, coughing, non-rigid patient movement during ENB procedure,and change in respiration patterns during the procedure. Each of theseactions may be modeled and utilized in accordance with aspects of thedisclosure.

A registration method is proposed that may use any of the deformationmodels described, any combination of them, or any other model, as longas the model can be empirically validated and the likelihood for eachmodeled deformation (referred to as a registration score) can bededuced. For example, it is less probable for an airway to extend twicein size from the time of the CT scan to the time of the ENB procedurethan for it to extend only 10%. Therefore, the probability of extensioncan be modeled to decrease with increase in degree of extension.

During an ENB procedure, the system uses the registration and/ornavigation samples to track catheter/bronchoscope/tool movement in theairways such that each sample can be matched to a specific point in thebronchial tree. This matching is based on the selected deformationmodel. Conventional lumen registration methods attempt to find a matchsuch that the mean distance between the survey and the bronchial tree isminimal. The conventional methods don't utilize the fact that the surveyhas to be continuous and follow a certain path through the lungs.Therefore, conventional registration methods may match neighbor samplesto completely separate segments in the lung, which reduces accuracy ofregistration. In contrast, the proposed method of the disclosure followsthe route of the tracked tool, sample by sample, from the trachea toother regions in the lung, thereby ensuring that a logical path is beingfollowed. Before samples are matched, filters that dilute the samples,remove outliers, and/or reduce respiration noise can be applied. Due toperformance considerations, a more detailed model can be used forsamples of interest and a rougher model can be used for other samples.

Since, using a given deformation model, there are usually multiple waysto match samples to the bronchial tree, the system selects those matchesin which the series of deformations that produce the match is mostprobable according to the model. For example, a deformation in which theright upper lobe has rotated 20 degrees relative to the right mainbronchus has a higher probability than a series of deformations in whichall five lobes have rotated 15 degrees. A registration is then createdbased on the selected match. Depending at least in part on thedeformation model, the registration can be flexible, such as a thinplate spline, it can consist of multiple rigid registrations, one foreach deformed region, or it can include a different registration foreach sample. The system may store in memory the top matches, so that ifa new sample or samples are received during navigation, the system canupdate the top matches and avoid re-computing all possible matches, manyof which may be improbable.

An example of a detailed implementation follows. A lumen registration isperformed. For each new navigation sample received by the system, thesystem tries to match the sample to the most likely airways in thebronchial tree. (see FIG. 5 , illustrating a match). This operation isperformed for each active match.

If multiple matches are likely, each match is evaluated and trackedindependently throughout navigation (see FIGS. 6A-6B, illustratingbranching). After subsequent navigation, each match can be split furtherto multiple matches (see FIGS. 6C-6D, illustrating further branching).Each active match is evaluated for fit by calculating the distance ofthe samples to the matched bronchial path (both position and orientationare used). Identification of a bifurcation depends on the angle betweenbranches and/or the length of each branch.

In addition, anchor points from the initial lumen registration are usedto constrain the registration (see FIG. 4 , illustrating registration).Constraining registration avoids excessive flexibility in theregistration process. For example, certain lobes can move by someamount, but some lobes may only stretch, and others only compress.Additionally, the lobes deformability may be interdependent. Forexample, if one region stretches 20%, other region may compress 15%, ormove and/or rotate in a similarly constrained fashion.

Significantly unfit matches are discarded. The fittest match is selectedto be used for final display of the endoscopic tool on a CT scan image.As new navigation samples are acquired, old navigation samples arediscarded in order to allow deformability (see FIG. 7 , illustratingbounding box). This bounding box may move with the catheter or othertool and may define a zone in which rigid registration is performed. Inthis manner, the model may be rigid in localized regions, but deformoutside that region by virtue of the flexibility of the alignmentbetween the different rigid registrations. Outlier samples may bediscarded assuming cough or tissue deformation by the catheter.Bifurcation areas are treated differently by delaying the splitting ofthe match until further matches are acquired. Transitions betweenbranches are smoothed to better represent real anatomy. Outside of thebronchial tree, a more conservative recent correction is used.

In some embodiments, image data may be acquired to generate, and bedisplayed in conjunction with or alongside, a digital reconstruction,such as a three-dimensional (3D) model or map, of the patient's lungs oranother portion of the patient's body. Various imaging modalities may beused to acquire the image data, including computed tomography (CT)imaging, cone beam computed tomography (CBCT) imaging, magneticresonance (MR) imaging, positron emission tomography (PET) imaging,fluoroscopic imaging, X-ray imaging, and/or ultrasound imaging. The 3Dmodel may be constructed based on preoperative image data from one ormore of the aforementioned imaging modalities. Alternatively oradditionally, additional image data, such as from a CBCT scan, may beacquired at the start of the treatment procedure and be used forregistration purposes, as further described below, and for constructingand/or enhancing the 3D model.

To create the 3D model, a preoperative segmental and subsegmentaldelineation and extrapolation may be performed based on image data ofthe patient's lungs to create a visual representation of the patient'slungs. The visual representation may include lumens, pleural surfaces,and fissures of the patient's lungs, and/or tumors or other aberrantstructures that may be present in the patient's lungs. The delineationmay be performed using one or more software applications executing on acomputer. The application may generate the 3D model of the patient'slungs based on the image data, noted above, to use for the visualrepresentation of the patient's lungs. The 3D model and image data maythen be viewed by a clinician and/or surgeon to plan a medical treatmentprocedure, such as a surgical or interventional procedure. The 3D modeland/or treatment plan may further be stored for later viewing during thetreatment procedure in an operating room or the like.

As described further below, the treatment plan may include identifiedlocations for one or more treatment targets, such as tumors, lesions, orother aberrant structures identified in the image data, and a pathwaybetween the patient's trachea and each of the treatment targets. Thepathway may include a portion located inside lumens, such as airways, ofthe patient's lungs, and a portion located outside of the airways of thepatient's lungs. An “exit point” may mark the transition point betweenthe portion of the pathway located inside the patient's airways and theportion of the pathway located outside of the patient's airways.

During the treatment procedure, the 3D model may be displayed, asfurther described below, to assist the clinician in navigating one ormore tools to the treatment target. The 3D model may include anindicator of a tracked position of the tool inside the patient's lungs.At various times during the treatment procedure, additional image datamay be acquired, such as by performing additional CBCT scans, to show areal-time location of the tool and/or the treatment target in thepatient's lungs. For example, after the tool passes the “exit point” andis located outside of the patient's airways, or at any other time of theclinician's choosing, additional image data may be acquired andprocessed to identify the tool and/or the treatment target. Theindicator on the 3D model of the tracked position of the tool may thenbe updated based on the additional image data, thereby showing aconfirmed location of the tool and/or the treatment target. Theadditional image data may further show, and thus enable a softwareapplication with the ability to track, the location of the tool duringvarious phases of the patient's respiration cycle. While the 3D modelmay be generated based on image data acquired while the patient was in aparticular phase of the respiration cycle, e.g. full breath hold, thepatient will not remain in that phase of the respiration cycle for theentire duration of the treatment procedure. Thus, acquiring image dataduring the treatment procedure during various phases of the patient'srespiration cycle, particularly during normal tidal volume breathing,may provide a clearer and more accurate visualization of the location ofthe tool and the treatment target inside the patient's lungs, as well asthe position of the tool relative to the treatment target.

Further, as will be appreciated by those skilled in the art, thedevices, systems, and methods described herein may also be used duringother types of medical procedures, such as percutaneous and/orlaparoscopic procedures, involving placement of a tool at a treatmentsite under image-guided and/or electromagnetic (EM) systems. As such,the illustrative embodiments described below are merely provided asexamples and are not intended to be limiting.

An electromagnetic navigation (EMN) system may be used for planning andperforming treatment of an area of a patient's lungs. Generally, in anembodiment, the EMN system may be used in planning treatment of an areaof the patient's lungs by identifying the positions of one or moretreatment targets in the patient's lungs, selecting one or more of thetreatment targets as a target location, determining a pathway to thetarget location, navigating a positioning assembly to the targetlocation, and navigating a variety of tools to the target location viathe positioning assembly. The EMN system may be configured to displayvarious views of the patient's lungs, including the aforementioned imagedata and 3D model.

FIG. 1 illustrates an EMN system 100 suitable for implementing methodsfor detecting tool displacement during medical procedures. EMN system100 is used to perform one or more procedures on a patient supported onan operating table 40 and generally includes monitoring equipment 30(e.g., video and/or image display), a bronchoscope 50, an EM trackingsystem 70, and a computing device 80.

Bronchoscope 50 is configured for insertion through the patient's mouthand/or nose into the patient's airways. Bronchoscope 50 includes asource of illumination and a video imaging system (not explicitly shown)and is coupled to monitoring equipment 30 for displaying the videoimages received from the video imaging system of bronchoscope 50. In anembodiment, bronchoscope 50 may operate in conjunction with a catheterguide assembly 90. Catheter guide assembly 90 includes a catheter 96configured for insertion through a working channel of bronchoscope 50into the patient's airways (although the catheter guide assembly 90 mayalternatively be used without bronchoscope 50). Catheter guide assembly90 further includes a handle 91 connected to catheter 96, and which canbe manipulated by rotation and compression to steer catheter 96 and/ortools inserted through catheter 96, such as a locatable guide (LG) 92.catheter 96 is sized for placement into the working channel ofbronchoscope 50. In the operation of catheter guide assembly 90, LG 92,including an EM sensor 94, is inserted into catheter 96 and locked intoposition such that EM sensor 94 extends a desired distance beyond adistal tip 93 of EWC 96. In some embodiments, catheter 96 may alsoinclude or have coupled thereto an EM sensor 95, and the EM sensors 94,95 may alternately or collectively be used to determine the position ofand steer catheter 96 and/or tools inserted therein. The location of EMsensors 94, 95, and thus distal tip 93 of catheter 96, within an EMfield generated by EM field generator 76, can be derived by trackingmodule 72 and computing device 80.

LG 92 and catheter 96 are selectively lockable relative to one anothervia a locking mechanism. A six degrees-of-freedom EM tracking system 70,or any other suitable positioning measuring system, is utilized forperforming navigation, although other configurations are alsocontemplated. Though sensors 94 and 95 are described herein as EMsensors, the disclosure is not so limited. The sensors 94 and 95 may beflexible sensors, such as fiber brag grating sensors, which can be usedto determine the flex and orientation of the catheter 96, or othersensors including ultrasound sensors, accelerometers, temperaturesensors, and others without departing from the scope of the disclosure.

EM tracking system 70 may be configured for use with catheter guideassembly 90 to track a position of EM sensors 94, 95 as they move inconjunction with catheter 96 through the airways of the patient. In anembodiment, EM tracking system 70 includes a tracking module 72, aplurality of reference sensors 74, and an EM field generator 76. Asshown in FIG. 1 , EM field generator 76 is positioned beneath thepatient. EM field generator 76 and the plurality of reference sensors 74are interconnected with tracking module 72, which derives the locationof each reference sensor 74 in the six degrees of freedom. One or moreof reference sensors 74 are attached to the chest of the patient. Thecoordinates of reference sensors 74 are sent as data to computing device80, which includes an application 81, where the data from referencesensors 74 are used to calculate a patient coordinate frame ofreference.

Although EM sensors 94, 95 are described above as being included in LG92 and catheter 96, respectively, an EM sensor may be embedded orincorporated within a treatment tool, such as a biopsy tool 62 and/or anablation tool 64, where the treatment tool may alternatively be utilizedfor navigation without need of LG 92 or the necessary tool exchangesthat use of LG 92 requires.

Treatment tools 62, 64 are configured to be insertable into catheterguide assembly 90 and catheter 96 following navigation to a targetlocation and removal of LG 92 (if used). Biopsy tool 62 may be used tocollect one or more tissue samples from the target location, and in anembodiment, is further configured for use in conjunction with trackingsystem 70 to facilitate navigation of biopsy tool 62 to the targetlocation, and tracking of a location of biopsy tool 62 as it ismanipulated relative to the target location to obtain the tissue sample.Ablation tool 64 is configured to be operated with a generator 66, suchas a radio frequency generator or a microwave generator and may includeany of a variety of ablation tools and/or catheters.

Though shown as a biopsy tool and microwave ablation tool in FIG. 1 ,those of skill in the art will recognize that other tools, including forexample RF ablation tools, brachytherapy tools, cryo-ablation tools, andothers may be similarly deployed and tracked without departing from thescope of the disclosure. Additionally, a piercing tool and/or puncturetool may be used and/or incorporated within LG 92 to create an exitpoint where LG 92, and thereby catheter 96, is navigated outside of thepatient's airways and toward the target location.

A radiographic imaging device 20, such as a computed tomography (CT)imaging device, magnetic resonance imaging (MM) imaging device, positronemission tomography (PET) imaging device, a cone beam computedtomography (CBCT) imaging device such as a C-arm imaging device, and/orany other imaging device capable of performing a scan of at least aportion of the patient's lungs, may be used in conjunction with EMNsystem 100. Imaging device 20 may further be capable of performingfluoroscopic scans of the patient's lungs. As shown in FIG. 1 , imagingdevice 20 is connected to computing device 80 such that application 81may receive and process image data obtained by imaging device 20.However, imaging device 20 may also have a separate computing devicelocated within the treatment room or in a separate control room to firstreceive the image data obtained by imaging device 20 and relay suchimage data to computing device 80. For example, to avoid exposing theclinician to unnecessary radiation from repeated radiographic scans, theclinician may exit the treatment room and wait in an adjacent room, suchas the control room, while imaging device 20 performs the scan.

Computing device 80 includes software and/or hardware, such asapplication 81, used to facilitate the various phases of an EMNprocedure, including generating a 3D model, identifying a targetlocation, planning a pathway to the target location, registering the 3Dmodel with the patient's actual airways, navigating to the targetlocation, and performing treatment at the target location. For example,computing device 80 utilizes data acquired from a CT scan, CBCT scan, MMscan, PET scan, and/or any other suitable imaging modality to generateand display the 3D model of the patient's airways, to enableidentification of a target location on the 3D model (automatically,semi-automatically or manually) by analyzing the image data and/or 3Dmodel, and allow for the determination and selection of a pathwaythrough the patient's airways to the target location. The 3D model maybe presented on a display monitor associated with computing device 80,or in any other suitable fashion.

Using computing device 80, various views of the image data and/or 3Dmodel may be displayed to and manipulated by a clinician to facilitateidentification of the target location. The target location may be a sitewithin the patient's lungs where treatment is to be performed. Forexample, the treatment target may be located in lung tissue adjacent toan airway. The 3D model may include, among other things, a model airwaytree corresponding to the actual airways of the patient's lungs, andshow the various passages, branches, and bifurcations of the patient'sactual airway tree. Additionally, the 3D model may includerepresentations of lesions, markers, blood vessels and vascularstructures, lymphatic vessels and structures, organs, otherphysiological structures, and/or a 3D rendering of the pleural surfacesand fissures of the patient's lungs. Some or all of the aforementionedelements may be selectively displayed, such that the clinician maychoose which elements should be displayed when viewing the 3D model.

After identifying the target location, application 81 may determine apathway between the patient's trachea and the target location via thepatient's airways. In instances where the target location is located inlung tissue that is not directly adjacent an airway, at least a portionof the pathway will be located outside of the patient's airways toconnect an exit point on an airway wall to the target location. In suchinstances, LG 94 and/or catheter 96 will first be navigated along afirst portion of the pathway through the patient's airways to the exitpoint on the airway wall. LG 94 may then be removed from catheter 96 andan access tool, such as a piercing or puncture tool, inserted intocatheter 96 to create an opening in the airway wall at the exit point.catheter 96 may then be advanced through the airway wall into theparenchyma surrounding the airways. The access tool may then be removedfrom catheter 96 and LG 94 and/or tools 62, 64 reinserted into catheter96 to navigate catheter 96 along a second portion of the pathway outsideof the airways to the target location.

During a procedure, EM sensors 94, 95, in conjunction with trackingsystem 70, enables tracking of EM sensors 94, 95 (and thus distal tip 93of catheter 96 or tools 62, 64) as catheter 96 is advanced through thepatient's airways following the pathway planned during the planningphase. As an initial step of the procedure, the 3D model is registeredwith the patient's actual airways to enable application 81 to display anindication of the position of EM sensors 94, 95 on the 3D modelcorresponding to the location of EM sensors 94, 95 within the patient'sairways.

One potential method of registration involves performing a survey of thepatient's lungs by navigating LG 92 into each lobe of the patient'slungs to at least the second bifurcation of the airways of that lobe.The position of LG 92 is tracked during this registration phase, and the3D model is iteratively updated based on the tracked position of LG 92within the actual airways of the patient's lungs. While the registrationprocess focuses on aligning the patient's actual airways with theairways of the 3D model, registration also ensures that the position ofvascular structures, pleural surfaces, and fissures of the lungs areaccurately determined. Anchor points 430 (FIG. 4 ), which may correspondto bifurcation points in the bronchial airway, may be obtained duringthis initial lumen registration and used to constrain the registration.

FIG. 2 illustrates a simplified block diagram of computing device 80.Computing device 80 may include a memory 202, a processor 204, a display206, a network interface 208, an input device 210, and/or an outputmodule 212. Memory 202 may store application 81 and/or image data 214.Application 81 may include instructions and/or executable code forgenerating a graphical user interface (GUI) 216 which, when executed byprocessor 204, cause display 206 to display a GUI.

Memory 202 may include any non-transitory computer-readable storagemedia for storing data and/or software that is executable by processor204 and which controls the operation of computing device 80. In anembodiment, memory 202 may include one or more solid-state storagedevices such as flash memory chips. Alternatively or in addition to theone or more solid-state storage devices, memory 202 may include one ormore mass storage devices connected to the processor 204 through a massstorage controller (not shown) and a communications bus (not shown).Although the description of computer-readable media contained hereinrefers to a solid-state storage, it should be appreciated by thoseskilled in the art that computer-readable storage media can be anyavailable media that can be accessed by the processor 204. That is,computer readable storage media includes non-transitory, volatile andnon-volatile, removable and non-removable media implemented in anymethod or technology for storage of information such ascomputer-readable instructions, data structures, program modules orother data. For example, computer-readable storage media includes RAM,ROM, EPROM, EEPROM, flash memory or other solid state memory technology,CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computing device 80.

Network interface 208 may be configured to connect to a network such asa local area network (LAN) consisting of a wired network and/or awireless network, a wide area network (WAN), a wireless mobile network,a Bluetooth network, and/or the internet. Input device 210 may be anydevice by means of which a user may interact with computing device 80,such as, for example, a mouse, keyboard, foot pedal, touch screen,and/or voice interface. Output module 212 may include any connectivityport or bus, such as, for example, parallel ports, serial ports,universal serial busses (USB), or any other similar connectivity portknown to those skilled in the art.

FIG. 3 is a flow chart illustrating a method 300 for registering aluminal network to a 3D model of the luminal network. Method 300 beginsat operation 302, which includes navigating a luminal network with atool including a location sensor. From operation 302, the flow in method300 proceeds to operation 304, which includes receiving location dataassociated with the location sensor. From operation 304, the flow inmethod 300 proceeds to operation 306, which includes identifyingpotential matches in an image of the luminal network with the locationdata. From optional operation 306, the flow in method 300 proceeds tooperation 308, which includes assigning potential matches a registrationscore based on a distance between the potential match and the locationdata, and a deformation model applied to the image. From operation 308,the flow in method 300 proceeds to operation 310, which includesdisplaying the potential match having a highest registration score. Fromoperation 310, the flow in method 300 proceeds to the end circle. Theorder of operations shown in FIG. 3 is exemplary only, and operationsmay be performed in a different order.

FIG. 4 shows exemplary 3D model 400 of a bronchial airway 410 having abranch 420. Also shown in FIG. 4 are sensor data 440, 450 obtained froma location sensor, for instance an EM sensor on a bronchoscope. Alsoillustrated in FIG. 4 are anchor points 430, which may correspond tobifurcation points in the bronchial airway 410 forming branches, forinstance, branch 420. Anchor points 430 may be obtained during theinitial lumen registration and may be used to constrain theregistration.

FIG. 5 shows lumen registration 500 in which a system matches newsamples to bronchial airway 410 of the 3D model. Registration datum 440includes both a position and direction 510. Registration datum 530occurs after bifurcation 520 in bronchial airway 410 and assists indetermining into which branch the bronchoscope has traveled.

FIG. 6A illustrates registration using 3D model 600 at a branching ofbronchial airway 410. Sensor data 620 includes a position and direction,and the position may be outside initial airway 610. However, based on adeformation model, sensor data 620 may receive a high registrationscore, and therefore may be considered a best estimate of the positionof the location sensor of the bronchoscope. 3D model 600 may be adjustedbased on the deformation model. Initial airway 610 may end atbifurcation 630, which may indicate the beginning of branches 640 and645. Another sensor data 650 may indicate a position beyond bifurcation630, but between branches 640 and 645. Therefore, the system may give aweighted score assigning sensor data 650 to branch 640.

FIG. 6B illustrates registration according to the disclosure as shown inFIG. 6A, using 3D model 600 at a branching of bronchial airway 410 atbifurcation 630. Sensor data 620 may be registered to initial airway610, as discussed above. Based on another sensor data 650, the systemmay not be able to conclude in which of branches 640 and 645 thebronchoscope is traveling, as discussed above, and therefore, the systemmay give another weighted score assigning sensor data 650 to branch 645.

FIG. 6C illustrates registration according to the disclosure as shown inFIGS. 6A and 6B, using 3D model 600 at a further branching of bronchialairway 410 at bifurcation 635. Based on another sensor data 650 andfurther sensor data 655, the system may conclude the bronchoscope istraveling down branch 645, and not branch 640. This conclusion may berealized by assigning a higher weighted score to registration withbranch 645 than registration with branch 640. Further sensor data 655may indicate a position beyond bifurcation 635. Therefore, the systemmay give a weighted score assigning sensor data 650 to branch 660.

FIG. 6D illustrates registration according to the disclosure as shown inFIGS. 6A, 6B, and 6C, using 3D model 600 at a branching of bronchialairway 410 at bifurcation 635. Based on further sensor data 655 beingpositioned significantly away from branch 665, the system may assign avery low score to registration of further data 655 with branch 665.Additionally or alternatively, the system may identify a largerdifference in direction between further sensor data 655 and branch 665than between further sensor data 655 and branch 660. Consequently, thesystem may assign a higher registration score to branch 660 based on thedifference in direction data. Alternatively, the system may discard allpossibilities associated with further data 655 being registered withbranch 665. In this manner, branch 660 may arise as the best possiblefit for further sensor data 655.

FIG. 7 shows a lumen registration performed in which the system matchesnew samples to bronchial airway 410 of 3D model 700 within movingboundary box 710. Moving boundary box 710 may move with the position ofthe location sensor, which may be centered within moving boundary box710 or may be elsewhere within moving boundary box 710. As thebronchoscope or other tool having the location sensor is moved withinbronchial airway 410, moving boundary box 710 may also move. Movingboundary box 710 may move incrementally, or in steps so that successivepositions of moving boundary box 710 do not overlap or only overlapslightly. Within moving boundary box 710, the system may perform a firstrigid registration between sensor data 620 and 3D model 700, and aftermoving boundary box 710, a second rigid registration may be performed.Subsequently, the first and second rigid registrations may be stitchedtogether to form a single registration.

In FIG. 7 , sensor data 620 includes a position and direction, and theposition may be outside initial airway 610. Based on a deformationmodel, sensor data 620 may receive a high registration score, andtherefore may be considered a best estimate of the position of thelocation sensor of the bronchoscope. 3D model 700 may be adjusted basedon the deformation model. Initial airway 610 may end at bifurcation 630,which may indicate the beginning of branches 640 and 645. Another sensordata 650 may indicate a position beyond bifurcation 630, but betweenbranches 640 and 645. Therefore, the system may give a weighted scoreassigning sensor data 650 to branch 640.

A further aspect of the disclosure is described with reference to FIG. 8. In FIG. 8 a method is described to improve the relationship between a3D model and sensor data received from a position sensor (e.g., EMsensor 94, 95). In accordance with the method, a pre-operative CT imagedata set is acquired of the patient at step 802. Typically, thepre-operative CT image data set, if of the lungs, is acquired while thepatient is at full breath hold. Those of ordinary skill in the art willrecognize that other forms of imaging may be employed including CBCT,MM, fluoroscopy and others without departing from the scope of thedisclosure, the one requirement is that the imaging technology becapable of providing images from which a 3D model can be generated. Atstep 804 a 3D model of the airways is generated from the pre-operativeCT image data set. By acquiring the CT image data set at full breathhold, determination of the boundaries of the airways in the 3D model ismade easier because the tissue in the image can be more readilydistinguished from the airways. Prior to generating the 3D model atreatment plan and the identification of targets within the CT imagedata set to which a catheter is to be navigated within the patient canbe identified and pathways to these targets are established as part ofthe treatment plan described in greater detail above.

Once the 3D model is generated, the 3D model and treatment plan may beloaded at step 806, into computing device 80 for use by application 81in conjunction with EMN system 100. After loading of the 3D model andthe treatment plan, a survey of the patient's physiology can beundertaken at step 808. In the case of a lung survey, a catheter 96including a sensor 94, 95 can be inserted into the airways and driven tocollect data regarding the shape of the airways in each of the lobes ofthe lungs. In the case of an EM sensor 94 or 95 being used, as thecatheter 96 and sensor is navigated through the airways and into eachlobe of the lungs, EM data points are collected. These may be collectedat a high frequency (e.g., 100-500 data points per second). The resultis a point cloud which resembles the shape of the airways in the lungs.It is understood that in general all of these datapoints will beacquired from within the airways of the lungs.

The data acquired from the survey of the lung by the sensor 94 can beused by the application 81 to generate a 3D model of the airways of thepatient at step 810. This 3D model of the airways of the patient may bequite different from the 3D model generated from the pre-operative CTimage data. As an initial matter and as noted above the CT image data istypically acquired while the patient is at full breath hold (i.e.maximum inflation) of the lungs, whereas the patient is experiencingtidal volume breathing while the catheter 96 is navigated through theairways to conduct the survey.

Typically, the point cloud generated from the survey will be used by theapplication 81 to register the 3D model generated from the pre-operativeCT image data to the physiology of the patient. In this way, thecatheter 96 can be navigated through the airways of the patient withoutthe need of a catheter-based imaging system, but rather by following thepathway and treatment plan through the 3D model generated from thepre-operative CT image data. However, this registration is necessarilyinexact owing to the differences in position of the patient's lungsduring the capture of the CT image data and during the navigationprocedure.

The differences in position of the lungs has been repeatedly observed,and in particular when intra-procedure CBCT images are acquired. Usingthese intra-procedure CBCT images a plurality of data transforms havebeen developed. These transforms are developed based on a comparison ofthe position of certain aspects of the 3D model from the pre-operativeCT image data with the position of those same aspects in a 3D modelderived from the CBCT images. These aspects may be the locations ofbifurcations or other observable aspects of the image data sets. As willbe appreciated, the central airways are less likely to move a great dealbetween the two image data sets, however, the closer one comes to theperiphery of the lungs, the greater the opportunity for movement. Thus,the transforms may include a graduation from central to peripheral inthe lungs. These transforms are based on hundreds or thousands ofcomparisons of pre-operative CT image data sets and intra-procedure CBCTimages and other data about the patients. Other data may include itemssuch as height, weight, sex, age, prior smoking activities, diseasestate, body fat, chest diameter, lung capacity, breathing rate, andother data about the patient. From this data numerous transforms can bedeveloped to accommodate most potential patients. These transforms maybe stored in the computing device 80 and accessed by the application 81.

At step 312 the application applies one or more of the transforms to the3D model generated from the pre-operative CT image data and compares theresults to the 3D model generated from the point cloud from the surveydata collected from the airways of the patient. The transform thatresults in the closest match to the 3D model generated from the pointcloud is utilized to transform the 3D model from the pre-operative CTimage data set. This closest match is an approximation of the change inthe 3D model generated from the pre-procedure CT image data set toachieve a shape of the airways defined in the 3D model from survey data(e.g., the point cloud).

In some instances, scaling may be necessary for application of thetransform to a particular transform. This may be necessary to compensatefor differences in scale (e.g., position of the images in a frame, ordistance from the imaging device) from the acquisition of the CT imagedata set as compared to the scale of the images used to develop thetransform. The scaling allows the transforms to be used on a wide arrayof CT image data sets

At step 314, the transformed 3D model is registered to the patient usingthe point cloud data collected and navigation of the patient's airwaysto an identified target can commence. This navigation of the patient'sairways is now undertaken with heightened confidence in the relativeposition of the 3D model and the actual locations of the catheter 96within the airways of the patient. Effectively, the application of thetransform eliminates much of the CT-to-body divergence that isexperienced based on differences in timing, positing, and breathingattributes, etc. between when the CT image data is collected and whenthe survey or point cloud data is collected.

As described above with respect to FIG. 7 , the transform need not beapplied globally to the entirety of the pre-procedure CT image data set,but rather can be selectively applied to portions of the CT image dataset so that a best fit of the transforms can be applied to selectportions. In this way each lobe of the lung, or a separation of left andright lungs, or another division may be employed and a differenttransform applied to each of these sub-units of the lungs to achieve anever closer transform of the pre-procedure CT image data set and the 3Dmodel derived therefrom to the 3D model generated from the survey data(e.g., the point cloud) of the airways of the patient.

While several embodiments of the disclosure have been shown in thedrawings, it is not intended that the disclosure be limited thereto, asit is intended that the disclosure be as broad in scope as the art willallow and that the specification be read likewise. Therefore, the abovedescription should not be construed as limiting, but merely asexemplifications of particular embodiments. Those skilled in the artwill envision other modifications within the scope and spirit of theclaims appended hereto.

What is claimed is:
 1. A method of registering a luminal network to a 3Dmodel of the luminal network, the method comprising: identifying a firstregion of the luminal network and a second region of the luminalnetwork; receiving location data associated with a location sensor on atool navigating the luminal network; identifying a plurality ofpotential matches in the 3D model from the location data; assigning atleast one of the potential matches a registration score based on adeformation model applied to the 3D model, the deformation modelincluding applying a moving window from the first region to the secondregion of the luminal network, and performing a rigid registration inthe moving window; and displaying a potential match of the plurality ofpotential matches having a highest registration score.
 2. The method ofclaim 1, wherein the assigning of the registration score is furtherbased on at least one of: a distance between the potential match and thelocation data; or a difference in direction between location sensordirection data and a direction of the potential match.
 3. The method ofclaim 1, further comprising, after receiving further location data,reassigning the at least one of the potential matches an updatedregistration score based on the further location data.
 4. The method ofclaim 1, wherein anchor points in the luminal network are used toconstrain the registration.
 5. The method of claim 1, wherein potentialmatches of the plurality of potential matches assigned a lowregistration score are discarded.
 6. The method of claim 5, furthercomprising delaying discarding potential matches with low registrationscores when the potential matches are in a bifurcation area of the 3Dmodel.
 7. The method of claim 1, wherein the deformation modelcomprises: at least one of: modeling at least one of rotation,compression, extension, or bending for the first region and the secondregion independently; modeling at least one of rotation, compression,extension, or bending for the first region and the second region withadjacent regions having interdependence; or performing rigidregistration of the first and second regions to form first and secondrigid registrations, and stitching together the first and second rigidregistrations.
 8. The method of claim 1, wherein: the luminal network isa bronchial airway; and the applying the moving window from a trachea toa periphery of the bronchial airway.
 9. The method of claim 1, whereinthe deformation model comprises at least one of: weighing registrationfor central areas of the luminal network greater than peripheral areasof the luminal network; using deformation models having differentdeformation characteristics for at least two of a right lung, a leftlung, or each lung lobe of the luminal network; using a biomechanicalmodel of the luminal network based on finite image analysis; using adeformation model based on a specific disease; or using at least one ofa bronchoscope model and a catheter model to produce a specificdeformation model.
 10. A system for updating a 3D model of a luminalnetwork, the system comprising: a location sensor on a tool configuredto be navigated within a luminal network inside a patient's body; an EMnavigation system including: an EM field generator configured togenerate an EM field; and an EM tracking system configured to detect alocation of the location sensor within the EM field; and a computingdevice including a processor and a memory storing instructions which,when executed by the processor, cause the computing device to: identifya first region of the luminal network and a second region of the luminalnetwork; receive location data associated with the location sensor;identify a plurality of potential matches in the 3D model from thelocation data; assign at least one of the plurality of potential matchesa registration score based on a deformation model applied to the 3Dmodel, the deformation model including applying a moving window from thefirst region to the second region of the luminal network, and performinga rigid registration in the moving window; and display a potential matchof the plurality of potential matches having a highest registrationscore.
 11. The system of claim 10, wherein the assigning of theregistration score is further based on at least one of: a distancebetween the potential match and the location data; or a difference indirection between location sensor direction data and a direction of thepotential match.
 12. The system of claim 10, wherein the instructionsfurther cause the computing device to, after receiving further locationdata, reassign the at least one of the potential matches an updatedregistration score based on the further location data.
 13. The system ofclaim 10, wherein the instructions further cause the computing device toupdate the 3D model with a plurality of potential matches having thehighest registration score.
 14. The system of claim 10, wherein thedeformation model comprises: at least one of: modeling at least one ofrotation, compression, extension, or bending for the first region andthe second region independently; modeling at least one of rotation,compression, extension, or bending for the first region and the secondregion with adjacent regions having interdependence; or performing rigidregistration of the first and second regions to form first and secondrigid registrations, and stitching together the first and second rigidregistrations.
 15. The system of claim 10, wherein the luminal networkincludes lungs and the deformation model comprises at least one of:weighing registration for central areas of the lungs greater thanperipheral areas of the lungs; using deformation models having differentdeformation characteristics for at least two of a right lung, a leftlung, or each lung lobe; using a biomechanical model of the lungs basedon finite image analysis; using a deformation model based on a specificdisease; or using at least one of a bronchoscope model or a cathetermodel to produce a specific deformation model.
 16. A method ofregistering a 3D model to a luminal network comprising: generating a 3Dmodel of a luminal network from a pre-procedure image data set;identifying a first region of the luminal network and a second region ofthe luminal network; conducting a survey of the luminal network;generating a 3D model of the luminal network from data captured duringthe survey; applying a transform to the 3D model of the luminal networkgenerated from the pre-procedure image data set to approximate the 3Dmodel of the luminal network generated from the data captured during thesurvey; wherein the transform model includes applying a moving windowfrom the first region to the second region of the luminal network, andperforming a rigid registration in the moving window.
 17. The method ofclaim 16, wherein the survey data is collected with a sensor insertedinto each lobe of a lung of a patient.
 18. The method of claim 17,wherein the sensor is an electromagnetic sensor collectingelectromagnetic position data.
 19. The method of claim 16, furthercomprising applying a plurality of transforms to the 3D model of theluminal network generated from the pre-procedure image data set.
 20. Themethod of claim 16, wherein the transform is selected by determiningwhich of a plurality of transforms result in a best fit of the 3D modelgenerated from the pre-procedure image data to the 3D model from thedata captured during the survey.